chore(base-image): Migrate Konflux builds to UBI9/RHEL9 by davdhacs · Pull Request #2562 · stackrox/scanner

davdhacs · 2026-01-15T04:28:55Z

Description

Migrates scanner builds from UBI8/RHEL8 to UBI9/RHEL9 base images.

Key Changes

Konflux Base Images:

Builder: brew.registry.redhat.io/rh-osbs/openshift-golang-builder:rhel_9_1.24
Scanner runtime: registry.access.redhat.com/ubi9-minimal:latest
Scanner DB: registry.redhat.io/rhel9/postgresql-15:latest
Image names: rhacs-scanner-rhel9, rhacs-scanner-slim-rhel9, rhacs-scanner-db-rhel9, rhacs-scanner-db-slim-rhel9

Non-Konflux Base Images:

Scanner: ubi9-minimal
Scanner DB: ubi9 / ubi9-minimal
Vulnerabilities: ubi9-minimal

RPM/Repo Updates:

rpms.lock.yaml: xz updated from RHEL 8 (5.2.4) to RHEL 9 (5.2.5)
rpms.rhel.repo: repos updated from rhel8 to rhel9
PostgreSQL download script: pg_rhel_major=9

Tekton Labels:

CPE labels: el8 → el9

UBI9 Compatibility Fixes:

update-ca-trust extract -o /etc/pki/ca-trust/extracted for unprivileged containers (RHBZ#2241240)
cp --recursive --no-dereference --no-clobber in restore-all-dir-contents
microdnf install -y xz (explicit -y flag)

Checklist

Investigated and inspected CI test results

Testing Performed

TBD

openshift-ci · 2026-01-15T04:29:19Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

davdhacs · 2026-01-27T14:57:14Z

@tommartensen fyi this is the parallel scanner update for UBI9. In this, we also have to add the update-ca-trust workaround for UBI9 changed perms on second execution.

The arm build fails because tar hits a bug in a syscall in qemu; I still have some debug logging around that, and I expect the arm arch build to fail because of it.

davdhacs · 2026-02-18T05:39:24Z

/test all

Migrate all scanner base images from UBI8/RHEL8 to UBI9/RHEL9: Konflux base images: - Builder: openshift-golang-builder:rhel_9_golang_1.25 - Scanner runtime: ubi9-minimal - Scanner DB: rhel9/postgresql-15 Non-Konflux base images: - Scanner: ubi9-minimal - Scanner DB: ubi9 / ubi9-minimal - Vulnerabilities: ubi9-minimal Other updates: - Tekton CPE labels: el8 -> el9 - rpms.lock.yaml: xz updated to RHEL 9 version (5.2.5-8.el9_0) - rpms.rhel.repo: RHEL 8 repos -> RHEL 9 repos - PostgreSQL download script: pg_rhel_major=9 - Image name labels: rhel8 -> rhel9 - microdnf install -y xz (explicit -y for UBI9 compatibility) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

In UBI9, update-ca-trust fails when running as an unprivileged user (nobody:nobody) because it attempts to write to system-wide paths. Use the -o flag to specify a user-writable output directory. Also switch restore-all-dir-contents to use --no-clobber to avoid overwriting CA trust files that were already updated at runtime. See: https://bugzilla.redhat.com/show_bug.cgi?id=2241240 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

davdhacs · 2026-02-18T21:32:25Z

/retest

davdhacs · 2026-02-19T00:07:01Z

@tommartensen could you review this? (the ubi9 upgrade for scanner(v2); includes the update-ca-trust permissions workaround)

tommartensen

LGTM to me from a Konflux perspective. Have you deployed and smoke tested the resulting images (like we did for collector)?

davdhacs · 2026-02-19T14:44:43Z

LGTM to me from a Konflux perspective. Have you deployed and smoke tested the resulting images (like we did for collector)?

Yes, I tested with these changes (minus the master-merge [empty] commits). I'll re-do the smoke test with this and the latest master collector builds.

BradLugo · 2026-02-23T17:06:39Z

Note to self and @stackrox/scanner: we need to follow up on #2092 and update the image for s390x to avoid using a different version of psql.

image/scanner/scripts/import-additional-cas

…migration

…migration # Conflicts: # image/db/rhel/konflux.Dockerfile

davdhacs · 2026-03-09T16:32:46Z

/test ?

davdhacs · 2026-03-10T04:25:22Z

The test failures are not-related to UBI9/changes in this PR.

davdhacs · 2026-03-10T20:51:26Z

/retest e2e-tests

davdhacs · 2026-03-10T23:33:48Z

/test e2e-tests

davdhacs · 2026-03-11T01:06:46Z

/test e2e-tests

davdhacs · 2026-03-11T03:47:19Z

/test e2e-tests

davdhacs · 2026-03-11T19:16:55Z

/test e2e-tests

davdhacs · 2026-03-11T21:09:17Z

@BradLugo can I get another +1? I merged changes from master to get the e2e test fix.

mclasmeier · 2026-03-16T11:29:15Z

I just did some testing with this scanner and I believe it suffers from the same problem as the one I have addressed with this commit.

See:

I am deploying ACS (some semi-recent master version, doesn't mater) on an OpenShift cluster, with scanner v2 enabled, scanner v4 disabled and an overlay which injects this PR's scanner version image into the scanner deployment:

❯ roxie deploy --resources=auto central \
  --set spec.scanner.scannerComponent=Enabled \
  --set spec.scannerV4.scannerComponent=Disabled \
  --set spec.overlays='[{"apiVersion":"apps/v1", "kind":"Deployment", "name":"scanner", "patches":[{"path":"spec.template.spec.containers[name:scanner].image", "value":"quay.io/rhacs-eng/scanner:2.39.x-86-gf65ee678a8"}]}]'
00:00 Using kubeconfig /Users/mclasmeier/.kube/infra/kubeconfig
00:00 Running with a controlling terminal.
00:01 🚀 ACS Deployer initialized
00:01 roxctl version: 4.9.1-0-g1bde1c73b2
00:01 
00:01 ┌────────────────────────────────────────────────────────────┐
00:01 │ Deployment Configuration                                   │
00:01 ├────────────────────────────────────────────────────────────┤
00:01 │              Component: Central                            │
00:01 │           Cluster Type: OpenShift4                         │
00:01 │               Main Tag:                                    │
00:01 │     Kubernetes Context: admin                              │
00:01 │      Deployment Method: Operator                           │
00:01 │               Exposure: loadbalancer                       │
00:01 └────────────────────────────────────────────────────────────┘
00:01 
00:01 Looking up main image tag
00:07 Using stackrox repository tag: 4.11.x-300-g280a4287f7
00:07 Auto-detected cluster type OpenShift4: using resource profile "medium"
00:07 Initiating deployment of Central
00:07 Deploying Central to namespace acs-central
00:07 Existing Central deployment found, tearing down...
00:07 🗑️  Tearing down acs-central
00:08 ⏳ Waiting for Central resources to be fully deleted...
00:08 Deleting Central resources
00:14 ✓ Central resources in namespace acs-central have been deleted
00:14 🚀 Deploying Central via Operator...
00:17 ✓ Operator already deployed with correct version
00:17 Preparing namespace acs-central
00:18 ✓ Admin password secret created
00:18 Using Central resource profile: medium
00:18 Applying Central custom resource
00:19 ✓ Central Custom Resource applied
00:19 ⏳ Waiting for Central to become ready...
[...]
01:00 ✓ Central is ready (1 replicas)
01:00 ⏳ Waiting for LoadBalancer central-loadbalancer to get external IP...
[...]
01:49 ✓ Central is ready and responding!
[...]

Then, I exec into the scanner container and show that the restore-all-dir-contents script cannot be invoked again (after the import-additional-cas script has run) -- something that I believe needs to be possible, because that would happen on container restarts.

❯ kubectl -n acs-central exec -it deploy/scanner -- /bin/sh
sh-5.1$ /restore-all-dir-contents 
cp: cannot create regular file '/etc/pki/ca-trust/extracted/pem/directory-hash/GlobalSign_ECC_Root_CA_-_R4.pem': Permission denied
cp: cannot create regular file '/etc/pki/ca-trust/extracted/pem/directory-hash/GlobalSign_ECC_Root_CA_-_R5.pem': Permission denied
cp: cannot create regular file '/etc/pki/ca-trust/extracted/pem/directory-hash/GlobalSign_Root_CA_-_R3.pem': Permission denied
cp: cannot create regular file '/etc/pki/ca-trust/extracted/pem/directory-hash/GlobalSign_Root_CA_-_R6.pem': Permission denied
cp: cannot create regular file '/etc/pki/ca-trust/extracted/pem/directory-hash/certSIGN_Root_CA_G2.pem': Permission denied
sh-5.1$

davdhacs · 2026-03-16T14:22:09Z

I just did some testing with this scanner and I believe it suffers from the same problem as the one I have addressed with this commit.

01:49 ✓ Central is ready and responding!

cp: cannot create regular file '/etc/pki/ca-trust/extracted/pem/directory-hash/GlobalSign_ECC_Root_CA_-_R5.pem': Permission denied

This looks like the chmod within the restore-* script is conclusively needed then? I'm out this week--could you commit the fix to this branch?

mclasmeier · 2026-03-17T11:05:43Z

I just did some testing with this scanner and I believe it suffers from the same problem as the one I have addressed with this commit.

01:49 ✓ Central is ready and responding!

cp: cannot create regular file '/etc/pki/ca-trust/extracted/pem/directory-hash/GlobalSign_ECC_Root_CA_-_R5.pem': Permission denied

This looks like the chmod within the restore-* script is conclusively needed then? I'm out this week--could you commit the fix to this branch?

Before I do that, let me run some more tests first, I'd like to check if we can address this ca-trust/save/restore issue a bit cleaner.

mclasmeier · 2026-03-18T10:37:37Z

Before I do that, let me run some more tests first, I'd like to check if we can address this ca-trust/save/restore issue a bit cleaner.

FTR, currently testing this even simpler fix:
stackrox/stackrox@247a365

Looking good so far.

Align the scanner entrypoint scripts and Dockerfiles with the proven CA trust setup from stackrox/stackrox#19454: - save-dir-contents: Only save /etc/pki/ca-trust/source (not the full /etc/pki/ca-trust or /etc/ssl). The extracted dir is regenerated at runtime by update-ca-trust. - Remove /etc/ssl from chown and save-dir-contents (not needed). - import-additional-cas: Add logging, use --output flag with detailed comment explaining why it's needed (BZ#2241240). - trust-root-ca: Same --output flag and comment. - restore-all-dir-contents: Add comment explaining --no-clobber. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

davdhacs · 2026-03-25T18:00:06Z

/retest-failed-builds

davdhacs · 2026-03-25T19:05:29Z

/test e2e-tests

…migration

…anner into konflux-ubi9-rhel9-migration

davdhacs · 2026-03-25T23:03:43Z

/retest

image/scanner/scripts/trust-root-ca

…anner into konflux-ubi9-rhel9-migration

davdhacs · 2026-03-27T13:58:53Z

/test e2e-tests

github-actions · 2026-04-01T13:54:34Z

/retest scanner-db-on-push

openshift-ci · 2026-04-01T13:54:44Z

@github-actions[bot]: The /retest command does not accept any targets.
The following commands are available to trigger optional jobs:

/test e2e-tests

/test slim-e2e-tests

Use /test all to run all jobs.

Details

In response to this:

/retest scanner-db-on-push

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

davdhacs added the konflux-build Run Konflux in PR. Push commit to trigger it. label Jan 15, 2026

openshift-ci bot added the do-not-merge/work-in-progress label Jan 15, 2026

davdhacs force-pushed the konflux-ubi9-rhel9-migration branch from d9cdc31 to d2f149d Compare February 5, 2026 16:58

davdhacs force-pushed the konflux-ubi9-rhel9-migration branch from 34c5d67 to 5a2e599 Compare February 18, 2026 05:37

davdhacs marked this pull request as ready for review February 18, 2026 05:39

davdhacs requested review from a team and rhacs-bot as code owners February 18, 2026 05:39

openshift-ci bot removed the do-not-merge/work-in-progress label Feb 18, 2026

rhacs-bot requested a review from a team February 18, 2026 05:48

davdhacs and others added 2 commits February 18, 2026 14:01

davdhacs force-pushed the konflux-ubi9-rhel9-migration branch from 5a2e599 to b3addf3 Compare February 18, 2026 21:07

empty commit

46d33f6

davdhacs requested a review from tommartensen February 19, 2026 00:06

tommartensen reviewed Feb 19, 2026

View reviewed changes

BradLugo approved these changes Feb 23, 2026

View reviewed changes

mclasmeier reviewed Mar 3, 2026

View reviewed changes

image/scanner/scripts/import-additional-cas Outdated Show resolved Hide resolved

davdhacs added 2 commits March 5, 2026 13:22

Merge remote-tracking branch 'origin/master' into konflux-ubi9-rhel9-…

1ec1ed6

…migration

Merge remote-tracking branch 'origin/master' into konflux-ubi9-rhel9-…

64b335a

…migration # Conflicts: # image/db/rhel/konflux.Dockerfile

davdhacs requested a review from BradLugo March 11, 2026 21:08

mclasmeier mentioned this pull request Mar 18, 2026

ROX-33603: Switch to UBI9 base images stackrox/stackrox#19454

Merged

4 tasks

davdhacs added 3 commits March 25, 2026 14:34

Merge branch 'master' into konflux-ubi9-rhel9-migration

ea5e1fe

Merge remote-tracking branch 'origin/master' into konflux-ubi9-rhel9-…

0bbcdeb

…migration

Merge branch 'konflux-ubi9-rhel9-migration' of github.com:stackrox/sc…

f48b1c7

…anner into konflux-ubi9-rhel9-migration

davdhacs commented Mar 25, 2026

View reviewed changes

image/scanner/scripts/trust-root-ca Show resolved Hide resolved

davdhacs added 2 commits March 25, 2026 18:42

Merge branch 'konflux-ubi9-rhel9-migration' of github.com:stackrox/sc…

d795c42

…anner into konflux-ubi9-rhel9-migration

Merge branch 'konflux-ubi9-rhel9-migration' of github.com:stackrox/sc…

d285ba8

…anner into konflux-ubi9-rhel9-migration

davdhacs requested a review from mclasmeier March 26, 2026 00:44

davdhacs force-pushed the konflux-ubi9-rhel9-migration branch from 5c49c9f to d285ba8 Compare March 26, 2026 21:46

restore before refactor

8222ac2

davdhacs requested a review from vladbologa April 1, 2026 13:41

Merge branch 'master' into konflux-ubi9-rhel9-migration

6f8d323

Conversation

davdhacs commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Key Changes

Checklist

Testing Performed

Uh oh!

openshift-ci bot commented Jan 15, 2026

Uh oh!

davdhacs commented Jan 27, 2026

Uh oh!

davdhacs commented Feb 18, 2026

Uh oh!

davdhacs commented Feb 18, 2026

Uh oh!

davdhacs commented Feb 19, 2026

Uh oh!

tommartensen left a comment

Choose a reason for hiding this comment

Uh oh!

davdhacs commented Feb 19, 2026

Uh oh!

BradLugo commented Feb 23, 2026

Uh oh!

Uh oh!

davdhacs commented Mar 9, 2026

Uh oh!

davdhacs commented Mar 10, 2026

Uh oh!

davdhacs commented Mar 10, 2026

Uh oh!

davdhacs commented Mar 10, 2026

Uh oh!

davdhacs commented Mar 11, 2026

Uh oh!

davdhacs commented Mar 11, 2026

Uh oh!

davdhacs commented Mar 11, 2026

Uh oh!

davdhacs commented Mar 11, 2026

Uh oh!

mclasmeier commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davdhacs commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mclasmeier commented Mar 17, 2026

Uh oh!

mclasmeier commented Mar 18, 2026

Uh oh!

davdhacs commented Mar 25, 2026

Uh oh!

davdhacs commented Mar 25, 2026

Uh oh!

davdhacs commented Mar 25, 2026

Uh oh!

Uh oh!

davdhacs commented Mar 27, 2026

Uh oh!

github-actions bot commented Apr 1, 2026

Uh oh!

openshift-ci bot commented Apr 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

davdhacs commented Jan 15, 2026 •

edited

Loading

mclasmeier commented Mar 16, 2026 •

edited

Loading

davdhacs commented Mar 16, 2026 •

edited

Loading